{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the Twitter Emotion Detection dataset using Autolabel"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set up the API keys for the providers you want to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# provide your own OpenAI API key here\n",
    "os.environ['OPENAI_API_KEY'] = 'sk-xxxxxxxxxxxxxxxxx'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Install the autolabel library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install 'refuel-autolabel[openai]'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-28 15:10:47 autolabel.utils ERROR: twitter_emotion_detection not in list of available datasets: ['banking', 'civil_comments', 'ledgar', 'walmart_amazon', 'company', 'squad_v2', 'sciq', 'conll2003', 'movie_reviews']. Exiting...\n"
     ]
    }
   ],
   "source": [
    "from autolabel import get_data\n",
    "\n",
    "get_data('twitter_emotion_detection')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This downloads two datasets:\n",
    "* `test.csv`: This is the larger dataset we are trying to label using LLMs\n",
    "* `seed.csv`: This is a small dataset where we already have human-provided labels"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start the labeling process!\n",
    "\n",
    "Labeling with Autolabel is a 3-step process:\n",
    "* First, we specify a labeling configuration (see `config.json` below)\n",
    "* Next, we do a dry-run on our dataset using the LLM specified in `config.json` by running `agent.plan`\n",
    "* Finally, we run the labeling with `agent.run`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from autolabel import LabelingAgent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = {\n",
    "    \"task_name\": \"EmotionClassification\",\n",
    "    \"task_type\": \"multilabel_classification\",\n",
    "    \"dataset\": {\"label_column\": \"labels\", \"label_separator\": \", \", \"delimiter\": \",\"},\n",
    "    \"model\": {\"provider\": \"openai\", \"name\": \"gpt-3.5-turbo\"},\n",
    "    \"prompt\": {\n",
    "        \"task_guidelines\": \"You are an expert at classifying tweets as neutral or one or more of the given emotions that best represent the mental state of the poster.\\nYour job is to correctly label the provided input example into one or more of the following categories:\\n{labels}\",\n",
    "        \"output_guidelines\": 'You will return the answer as a comma separated list of labels sorted in alphabetical order. For example: \"label1, label2, label3\"',\n",
    "        \"labels\": [\n",
    "            \"neutral\",\n",
    "            \"anger\",\n",
    "            \"anticipation\",\n",
    "            \"disgust\",\n",
    "            \"fear\",\n",
    "            \"joy\",\n",
    "            \"love\",\n",
    "            \"optimism\",\n",
    "            \"pessimism\",\n",
    "            \"sadness\",\n",
    "            \"surprise\",\n",
    "            \"trust\",\n",
    "        ],\n",
    "        \"few_shot_examples\": \"data/twitter_emotion_detection/seed.csv\",\n",
    "        \"few_shot_selection\": \"semantic_similarity\",\n",
    "        \"few_shot_num\": 5,\n",
    "        \"example_template\": \"Input: {example}\\nOutput: {labels}\",\n",
    "    },\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's review the configuration file below. You'll notice the following useful keys:\n",
    "* `task_type`: `multilabel_classification` (since it's a multi label classification task)\n",
    "* `model`: `{'provider': 'openai', 'name': 'gpt-3.5-turbo'}` (use a specific OpenAI model)\n",
    "* `prompt.labels`: `['neutral', 'anger', 'anticipation', 'disgust', 'fear', ...]` (the full list of labels to choose from)\n",
    "* `prompt.task_guidelines`: `'You are an expert at classifying tweets as...` (how we describe the task to the LLM)\n",
    "* `prompt.few_shot_num`: 5 (how many labeled examples to provide to the LLM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'task_name': 'EmotionClassification',\n",
       " 'task_type': 'multilabel_classification',\n",
       " 'dataset': {'label_column': 'labels',\n",
       "  'label_separator': ', ',\n",
       "  'delimiter': ','},\n",
       " 'model': {'provider': 'openai', 'name': 'gpt-3.5-turbo'},\n",
       " 'prompt': {'task_guidelines': 'You are an expert at classifying tweets as neutral or one or more of the given emotions that best represent the mental state of the poster.\\nYour job is to correctly label the provided input example into one or more of the following categories:\\n{labels}',\n",
       "  'output_guidelines': 'You will return the answer as a comma separated list of labels sorted in alphabetical order. For example: \"label1, label2, label3\"',\n",
       "  'labels': ['neutral',\n",
       "   'anger',\n",
       "   'anticipation',\n",
       "   'disgust',\n",
       "   'fear',\n",
       "   'joy',\n",
       "   'love',\n",
       "   'optimism',\n",
       "   'pessimism',\n",
       "   'sadness',\n",
       "   'surprise',\n",
       "   'trust'],\n",
       "  'few_shot_examples': 'seed.csv',\n",
       "  'few_shot_selection': 'semantic_similarity',\n",
       "  'few_shot_num': 5,\n",
       "  'example_template': 'Input: {example}\\nOutput: {labels}'}}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an aggent for labeling\n",
    "agent = LabelingAgent(config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b943d3ed3e16483a8acdc49c8bcc6366",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┌──────────────────────────┬─────────┐\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Total Estimated Cost     </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $4.9705 </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Number of Examples       </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> 2000    </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Average cost per example </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $0.0025 </span>│\n",
       "└──────────────────────────┴─────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┌──────────────────────────┬─────────┐\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mTotal Estimated Cost    \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$4.9705\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mNumber of Examples      \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m2000   \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mAverage cost per example\u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$0.0025\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "└──────────────────────────┴─────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────── </span>Prompt Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────── \u001b[0mPrompt Example\u001b[92m ──────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are an expert at classifying tweets as neutral or one or more of the given emotions that best represent the mental state of the poster.\n",
      "Your job is to correctly label the provided input example into one or more of the following categories:\n",
      "neutral\n",
      "anger\n",
      "anticipation\n",
      "disgust\n",
      "fear\n",
      "joy\n",
      "love\n",
      "optimism\n",
      "pessimism\n",
      "sadness\n",
      "surprise\n",
      "trust\n",
      "\n",
      "You will return the answer as a comma separated list of labels sorted in alphabetical order. For example: \"label1, label2, label3\"\n",
      "\n",
      "Some examples with their output answers are provided below:\n",
      "\n",
      "Input: @MaryamNSharif I think just becz u have so much terror in pak nd urself being  a leader u forgot d difference btw a leader nd terrorist !\n",
      "Output: anger, disgust, fear\n",
      "\n",
      "Input: In wake of fresh #terror threat and sounding of alert in #Mumbai, praying for safety &amp; security of everybody in the city #Maharashtra #news\n",
      "Output: fear\n",
      "\n",
      "Input: Somewhere I rd a rpt tht Pakis wr afraid of TSD &amp; asked it 2 shut dn. Congis obliged &amp; exposed it,hounded them.time to revive. #BadlaofUri\n",
      "Output: neutral\n",
      "\n",
      "Input: Although this war will be under the guise of combating terrorism it will in fact be a war against poverty, ignorance and just stupidity.\n",
      "Output: anger, disgust, optimism\n",
      "\n",
      "Input: @DaniQays @AJENews ohh.. so here comes sense from terror supporting stone pelting vandalizing ppl who gather b4 protests to announce ...\n",
      "Output: anger, disgust, fear\n",
      "\n",
      "Now I want you to label the following example:\n",
      "Input: @Adnan__786__ @AsYouNotWish Dont worry Indian army is on its ways to dispatch all Terrorists to Hell\n",
      "Output: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from autolabel import AutolabelDataset\n",
    "ds = AutolabelDataset(\"data/twitter_emotion_detection/test.csv\", config=config)\n",
    "agent.plan(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-28 15:11:19 autolabel.labeler INFO: Task run already exists.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">There is an existing task with following details: <span style=\"color: #808000; text-decoration-color: #808000\">id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'148569753'</span> <span style=\"color: #808000; text-decoration-color: #808000\">created_at</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">datetime</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">.datetime</span><span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2023</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>, \n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">34</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">979450</span><span style=\"font-weight: bold\">)</span> <span style=\"color: #808000; text-decoration-color: #808000\">task_id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'f3f9865e0ef223de46a41a00d30b0847'</span> <span style=\"color: #808000; text-decoration-color: #808000\">dataset_id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'6972c71134baa2a223a510daf02a6997'</span> \n",
       "<span style=\"color: #808000; text-decoration-color: #808000\">current_index</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span> <span style=\"color: #808000; text-decoration-color: #808000\">output_file</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'test_labeled.csv'</span> <span style=\"color: #808000; text-decoration-color: #808000\">status</span>=<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">TaskStatus.ACTIVE:</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">'active'</span><span style=\"font-weight: bold\">&gt;</span> <span style=\"color: #808000; text-decoration-color: #808000\">error</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span> <span style=\"color: #808000; text-decoration-color: #808000\">metrics</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "There is an existing task with following details: \u001b[33mid\u001b[0m=\u001b[32m'148569753'\u001b[0m \u001b[33mcreated_at\u001b[0m=\u001b[1;35mdatetime\u001b[0m\u001b[1;35m.datetime\u001b[0m\u001b[1m(\u001b[0m\u001b[1;36m2023\u001b[0m, \u001b[1;36m6\u001b[0m, \u001b[1;36m28\u001b[0m, \u001b[1;36m15\u001b[0m, \u001b[1;36m10\u001b[0m, \n",
       "\u001b[1;36m34\u001b[0m, \u001b[1;36m979450\u001b[0m\u001b[1m)\u001b[0m \u001b[33mtask_id\u001b[0m=\u001b[32m'f3f9865e0ef223de46a41a00d30b0847'\u001b[0m \u001b[33mdataset_id\u001b[0m=\u001b[32m'6972c71134baa2a223a510daf02a6997'\u001b[0m \n",
       "\u001b[33mcurrent_index\u001b[0m=\u001b[1;36m0\u001b[0m \u001b[33moutput_file\u001b[0m=\u001b[32m'test_labeled.csv'\u001b[0m \u001b[33mstatus\u001b[0m=\u001b[1m<\u001b[0m\u001b[1;95mTaskStatus.ACTIVE:\u001b[0m\u001b[39m \u001b[0m\u001b[32m'active'\u001b[0m\u001b[1m>\u001b[0m \u001b[33merror\u001b[0m=\u001b[3;35mNone\u001b[0m \u001b[33mmetrics\u001b[0m=\u001b[3;35mNone\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Evaluating the existing task<span style=\"color: #808000; text-decoration-color: #808000\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Evaluating the existing task\u001b[33m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/workrefuel/.pyenv/versions/3.8.16/envs/refuel/lib/python3.8/site-packages/sklearn/preprocessing/_label.py:895: UserWarning: unknown class(es) ['anxiety'] will be ignored\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> f1     </span>┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃\n",
       "┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1778 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 5       </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.0      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1.0             </span>│\n",
       "└────────┴─────────┴──────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mf1    \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.1778\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m5      \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.0     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1.0            \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└────────┴─────────┴──────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span> examples labeled so far.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;36m0\u001b[0m examples labeled so far.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────── </span>Last Annotated Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────── \u001b[0mLast Annotated Example\u001b[92m ──────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Prompt</span>: </pre>\n"
      ],
      "text/plain": [
       "\u001b[1;34mPrompt\u001b[0m: "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are an expert at classifying tweets as neutral or one or more of the given emotions that best represent the mental state of the poster.\n",
      "Your job is to correctly label the provided input example into one or more of the following categories:\n",
      "neutral\n",
      "anger\n",
      "anticipation\n",
      "disgust\n",
      "fear\n",
      "joy\n",
      "love\n",
      "optimism\n",
      "pessimism\n",
      "sadness\n",
      "surprise\n",
      "trust\n",
      "\n",
      "You will return the answer as a comma separated list of labels sorted in alphabetical order. For example: \"label1, label2, label3\"\n",
      "\n",
      "Some examples with their output answers are provided below:\n",
      "\n",
      "Input: #PeopleLikeMeBecause they see the happy exterior, not the hopelessness I sometimes feel inside. #depression #anxiety #anxietyprobz\n",
      "Output: fear, sadness\n",
      "\n",
      "Input: #PeopleLikeMeBecause they see the happy exterior, not the hopelessness I sometimes feel inside. #depression  #anxietyprobz\n",
      "Output: fear, sadness\n",
      "\n",
      "Input: Living with #depression doesn't mean you must be defeated by it\\nevery day's a new day and yesterday doesn't decide what today looks like :-)\n",
      "Output: optimism, sadness\n",
      "\n",
      "Input: Love is when all your happiness and all your sadness and all your feelings are dependent on another person.\n",
      "Output: love, optimism, trust\n",
      "\n",
      "Input: It’s possible changing meds is best not done while under stress. Difficult to tell what part of despair is circumstantial, what is drugs.\n",
      "Output: sadness\n",
      "\n",
      "Now I want you to label the following example:\n",
      "Input: #Deppression is real. Partners w/ #depressed people truly dont understand the depth in which they affect us. Add in #anxiety &amp;makes it worse\n",
      "Output: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Annotation</span>: </pre>\n"
      ],
      "text/plain": [
       "\u001b[1;34mAnnotation\u001b[0m: "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "anxiety, sadness\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Do you want to resume the task? <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">[y/n]</span>: </pre>\n"
      ],
      "text/plain": [
       "Do you want to resume the task? \u001b[1;35m[y/n]\u001b[0m: "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Deleted the existing task and starting a new one<span style=\"color: #808000; text-decoration-color: #808000\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Deleted the existing task and starting a new one\u001b[33m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "01cd831b4a8e48489b55b0f5a93b8b99",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">/Users/workrefuel/.pyenv/versions/3.8.16/envs/refuel/lib/python3.8/site-packages/sklearn/preprocessing/_label.py:89\n",
       "5: UserWarning: unknown class(es) ['anxiety', 'happiness', 'trauma'] will be ignored\n",
       "  warnings.warn(\n",
       "</pre>\n"
      ],
      "text/plain": [
       "/Users/workrefuel/.pyenv/versions/3.8.16/envs/refuel/lib/python3.8/site-packages/sklearn/preprocessing/_label.py:89\n",
       "5: UserWarning: unknown class(es) ['anxiety', 'happiness', 'trauma'] will be ignored\n",
       "  warnings.warn(\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">/Users/workrefuel/.pyenv/versions/3.8.16/envs/refuel/lib/python3.8/site-packages/sklearn/preprocessing/_label.py:89\n",
       "5: UserWarning: unknown class(es) ['anxiety', 'happiness', 'trauma'] will be ignored\n",
       "  warnings.warn(\n",
       "</pre>\n"
      ],
      "text/plain": [
       "/Users/workrefuel/.pyenv/versions/3.8.16/envs/refuel/lib/python3.8/site-packages/sklearn/preprocessing/_label.py:89\n",
       "5: UserWarning: unknown class(es) ['anxiety', 'happiness', 'trauma'] will be ignored\n",
       "  warnings.warn(\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual Cost: 0.0025\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/workrefuel/.pyenv/versions/3.8.16/envs/refuel/lib/python3.8/site-packages/sklearn/preprocessing/_label.py:895: UserWarning: unknown class(es) ['anxiety', 'happiness', 'trauma'] will be ignored\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> f1     </span>┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃\n",
       "┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4507 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 100     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.08     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1.0             </span>│\n",
       "└────────┴─────────┴──────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mf1    \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4507\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m100    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.08    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1.0            \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└────────┴─────────┴──────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Total number of failures: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Total number of failures: \u001b[1;36m0\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# now, do the actual labeling\n",
    "ds = agent.run(ds, max_items=100)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are at a 0.4507 macro f1 when labeling the first 100 examples. Let's see if we can use confidence scores to improve f1 further by removing the less confident examples from our labeled set."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute confidence scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start computing confidence scores (using Refuel's LLMs)\n",
    "os.environ['REFUEL_API_KEY'] = 'sk-xxxxxxxxxxxxxxxxx'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set `compute_confidence` -> True\n",
    "config[\"model\"][\"compute_confidence\"] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = LabelingAgent(config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "707d5cef986d4274bc7eba32b4792c31",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┌──────────────────────────┬─────────┐\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Total Estimated Cost     </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $4.9705 </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Number of Examples       </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> 2000    </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Average cost per example </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $0.0025 </span>│\n",
       "└──────────────────────────┴─────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┌──────────────────────────┬─────────┐\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mTotal Estimated Cost    \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$4.9705\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mNumber of Examples      \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m2000   \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mAverage cost per example\u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$0.0025\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "└──────────────────────────┴─────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────── </span>Prompt Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────── \u001b[0mPrompt Example\u001b[92m ──────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are an expert at classifying tweets as neutral or one or more of the given emotions that best represent the mental state of the poster.\n",
      "Your job is to correctly label the provided input example into one or more of the following categories:\n",
      "neutral\n",
      "anger\n",
      "anticipation\n",
      "disgust\n",
      "fear\n",
      "joy\n",
      "love\n",
      "optimism\n",
      "pessimism\n",
      "sadness\n",
      "surprise\n",
      "trust\n",
      "\n",
      "You will return the answer as a comma separated list of labels sorted in alphabetical order. For example: \"label1, label2, label3\"\n",
      "\n",
      "Some examples with their output answers are provided below:\n",
      "\n",
      "Input: @MaryamNSharif I think just becz u have so much terror in pak nd urself being  a leader u forgot d difference btw a leader nd terrorist !\n",
      "Output: anger, disgust, fear\n",
      "\n",
      "Input: In wake of fresh #terror threat and sounding of alert in #Mumbai, praying for safety &amp; security of everybody in the city #Maharashtra #news\n",
      "Output: fear\n",
      "\n",
      "Input: Somewhere I rd a rpt tht Pakis wr afraid of TSD &amp; asked it 2 shut dn. Congis obliged &amp; exposed it,hounded them.time to revive. #BadlaofUri\n",
      "Output: neutral\n",
      "\n",
      "Input: Although this war will be under the guise of combating terrorism it will in fact be a war against poverty, ignorance and just stupidity.\n",
      "Output: anger, disgust, optimism\n",
      "\n",
      "Input: @DaniQays @AJENews ohh.. so here comes sense from terror supporting stone pelting vandalizing ppl who gather b4 protests to announce ...\n",
      "Output: anger, disgust, fear\n",
      "\n",
      "Now I want you to label the following example:\n",
      "Input: @Adnan__786__ @AsYouNotWish Dont worry Indian army is on its ways to dispatch all Terrorists to Hell\n",
      "Output: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from autolabel import AutolabelDataset\n",
    "ds = AutolabelDataset(\"data/twitter_emotion_detection/test.csv\", config=config)\n",
    "agent.plan(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-28 15:12:59 autolabel.labeler INFO: Task run already exists.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">There is an existing task with following details: <span style=\"color: #808000; text-decoration-color: #808000\">id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'2545097544'</span> <span style=\"color: #808000; text-decoration-color: #808000\">created_at</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">datetime</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">.datetime</span><span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2023</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">14</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">36</span>,\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">629024</span><span style=\"font-weight: bold\">)</span> <span style=\"color: #808000; text-decoration-color: #808000\">task_id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'f1cc75949ce09a9b60ccfcd75bf6f7a4'</span> <span style=\"color: #808000; text-decoration-color: #808000\">dataset_id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'6972c71134baa2a223a510daf02a6997'</span> \n",
       "<span style=\"color: #808000; text-decoration-color: #808000\">current_index</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> <span style=\"color: #808000; text-decoration-color: #808000\">output_file</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'test_labeled.csv'</span> <span style=\"color: #808000; text-decoration-color: #808000\">status</span>=<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">TaskStatus.ACTIVE:</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">'active'</span><span style=\"font-weight: bold\">&gt;</span> <span style=\"color: #808000; text-decoration-color: #808000\">error</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span> <span style=\"color: #808000; text-decoration-color: #808000\">metrics</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "There is an existing task with following details: \u001b[33mid\u001b[0m=\u001b[32m'2545097544'\u001b[0m \u001b[33mcreated_at\u001b[0m=\u001b[1;35mdatetime\u001b[0m\u001b[1;35m.datetime\u001b[0m\u001b[1m(\u001b[0m\u001b[1;36m2023\u001b[0m, \u001b[1;36m6\u001b[0m, \u001b[1;36m28\u001b[0m, \u001b[1;36m14\u001b[0m, \u001b[1;36m36\u001b[0m,\n",
       "\u001b[1;36m8\u001b[0m, \u001b[1;36m629024\u001b[0m\u001b[1m)\u001b[0m \u001b[33mtask_id\u001b[0m=\u001b[32m'f1cc75949ce09a9b60ccfcd75bf6f7a4'\u001b[0m \u001b[33mdataset_id\u001b[0m=\u001b[32m'6972c71134baa2a223a510daf02a6997'\u001b[0m \n",
       "\u001b[33mcurrent_index\u001b[0m=\u001b[1;36m100\u001b[0m \u001b[33moutput_file\u001b[0m=\u001b[32m'test_labeled.csv'\u001b[0m \u001b[33mstatus\u001b[0m=\u001b[1m<\u001b[0m\u001b[1;95mTaskStatus.ACTIVE:\u001b[0m\u001b[39m \u001b[0m\u001b[32m'active'\u001b[0m\u001b[1m>\u001b[0m \u001b[33merror\u001b[0m=\u001b[3;35mNone\u001b[0m \u001b[33mmetrics\u001b[0m=\u001b[3;35mNone\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Evaluating the existing task<span style=\"color: #808000; text-decoration-color: #808000\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Evaluating the existing task\u001b[33m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metric: auroc: 0.7608695652173914\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/workrefuel/.pyenv/versions/3.8.16/envs/refuel/lib/python3.8/site-packages/sklearn/preprocessing/_label.py:895: UserWarning: unknown class(es) ['anxiety', 'happiness', 'trauma'] will be ignored\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> f1     </span>┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃<span style=\"font-weight: bold\"> threshold </span>┃\n",
       "┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.0833 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1       </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.0      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.01            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.9555    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.25   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 2       </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.5      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.02            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.9248    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4048 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 7       </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1429   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.07            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.8177    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4074 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 8       </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.25     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.08            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.8099    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4091 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 10      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.2      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1             </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.7784    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4119 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 11      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.2727   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.11            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.7746    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.3937 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 12      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.25     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.12            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.765     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.3997 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 13      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.3077   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.13            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.7606    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4279 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 29      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1379   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.29            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.6927    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4341 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 30      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1667   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.3             </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.6805    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4508 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 35      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1429   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.35            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.6213    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4537 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 36      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1667   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.36            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.6114    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4842 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 49      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1224   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.49            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.5561    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4932 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 50      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.14     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.5             </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.5553    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4629 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 61      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1148   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.61            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.5011    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4639 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 62      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.129    </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.62            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4988    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4507 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 100     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.08     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1.0             </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.0206    </span>│\n",
       "└────────┴─────────┴──────────┴─────────────────┴───────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mf1    \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mthreshold\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.0833\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1      \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.0     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.01           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.9555   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.25  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m2      \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.5     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.02           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.9248   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4048\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m7      \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1429  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.07           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.8177   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4074\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m8      \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.25    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.08           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.8099   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4091\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m10     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.2     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1            \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.7784   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4119\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m11     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.2727  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.11           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.7746   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.3937\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m12     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.25    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.12           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.765    \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.3997\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m13     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.3077  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.13           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.7606   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4279\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m29     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1379  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.29           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.6927   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4341\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m30     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1667  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.3            \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.6805   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4508\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m35     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1429  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.35           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.6213   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4537\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m36     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1667  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.36           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.6114   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4842\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m49     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1224  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.49           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.5561   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4932\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m50     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.14    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.5            \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.5553   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4629\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m61     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1148  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.61           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.5011   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4639\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m62     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.129   \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.62           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.4988   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4507\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m100    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.08    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1.0            \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.0206   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└────────┴─────────┴──────────┴─────────────────┴───────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span> examples labeled so far.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;36m100\u001b[0m examples labeled so far.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────── </span>Last Annotated Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────── \u001b[0mLast Annotated Example\u001b[92m ──────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Prompt</span>: </pre>\n"
      ],
      "text/plain": [
       "\u001b[1;34mPrompt\u001b[0m: "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are an expert at classifying tweets as neutral or one or more of the given emotions that best represent the mental state of the poster.\n",
      "Your job is to correctly label the provided input example into one or more of the following categories:\n",
      "neutral\n",
      "anger\n",
      "anticipation\n",
      "disgust\n",
      "fear\n",
      "joy\n",
      "love\n",
      "optimism\n",
      "pessimism\n",
      "sadness\n",
      "surprise\n",
      "trust\n",
      "\n",
      "You will return the answer as a comma separated list of labels sorted in alphabetical order. For example: \"label1, label2, label3\"\n",
      "\n",
      "Some examples with their output answers are provided below:\n",
      "\n",
      "Input: Sometimes I like to talk about my sadness.  Other times, I just want to be distracted by friends, laughter, shopping, eating...  \\n\\n#MHChat\n",
      "Output: disgust, sadness\n",
      "\n",
      "Input: #PeopleLikeMeBecause they see the happy exterior, not the hopelessness I sometimes feel inside. #depression #anxiety #anxietyprobz\n",
      "Output: fear, sadness\n",
      "\n",
      "Input: #PeopleLikeMeBecause they see the happy exterior, not the hopelessness I sometimes feel inside. #depression  #anxietyprobz\n",
      "Output: fear, sadness\n",
      "\n",
      "Input: Love is when all your happiness and all your sadness and all your feelings are dependent on another person.\n",
      "Output: love, optimism, trust\n",
      "\n",
      "Input: Whatever you decide to do make sure it makes you #happy.\n",
      "Output: joy, love, optimism\n",
      "\n",
      "Now I want you to label the following example:\n",
      "Input: There are parts of you that wants the sadness. Find them out, ask them why\n",
      "Output: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Annotation</span>: </pre>\n"
      ],
      "text/plain": [
       "\u001b[1;34mAnnotation\u001b[0m: "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sadness\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Do you want to resume the task? <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">[y/n]</span>: </pre>\n"
      ],
      "text/plain": [
       "Do you want to resume the task? \u001b[1;35m[y/n]\u001b[0m: "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Deleted the existing task and starting a new one<span style=\"color: #808000; text-decoration-color: #808000\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Deleted the existing task and starting a new one\u001b[33m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "726670e9003d4f54880a8bd43f362afb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">/Users/workrefuel/.pyenv/versions/3.8.16/envs/refuel/lib/python3.8/site-packages/sklearn/preprocessing/_label.py:89\n",
       "5: UserWarning: unknown class(es) ['anxiety', 'happiness', 'trauma'] will be ignored\n",
       "  warnings.warn(\n",
       "</pre>\n"
      ],
      "text/plain": [
       "/Users/workrefuel/.pyenv/versions/3.8.16/envs/refuel/lib/python3.8/site-packages/sklearn/preprocessing/_label.py:89\n",
       "5: UserWarning: unknown class(es) ['anxiety', 'happiness', 'trauma'] will be ignored\n",
       "  warnings.warn(\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-28 15:16:58 autolabel.confidence WARNING: Retrying autolabel.confidence.ConfidenceCalculator._call_with_retry in 2.0 seconds as it raised HTTPError: 429 Client Error: Too Many Requests for url: https://refuel-llm.refuel.ai/.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-28 15:17:02 autolabel.confidence WARNING: Retrying autolabel.confidence.ConfidenceCalculator._call_with_retry in 2.0 seconds as it raised HTTPError: 429 Client Error: Too Many Requests for url: https://refuel-llm.refuel.ai/.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">/Users/workrefuel/.pyenv/versions/3.8.16/envs/refuel/lib/python3.8/site-packages/sklearn/preprocessing/_label.py:89\n",
       "5: UserWarning: unknown class(es) ['anxiety', 'happiness', 'trauma'] will be ignored\n",
       "  warnings.warn(\n",
       "</pre>\n"
      ],
      "text/plain": [
       "/Users/workrefuel/.pyenv/versions/3.8.16/envs/refuel/lib/python3.8/site-packages/sklearn/preprocessing/_label.py:89\n",
       "5: UserWarning: unknown class(es) ['anxiety', 'happiness', 'trauma'] will be ignored\n",
       "  warnings.warn(\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metric: auroc: 0.7609\n",
      "Actual Cost: 0.0025\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/workrefuel/.pyenv/versions/3.8.16/envs/refuel/lib/python3.8/site-packages/sklearn/preprocessing/_label.py:895: UserWarning: unknown class(es) ['anxiety', 'happiness', 'trauma'] will be ignored\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> f1     </span>┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃<span style=\"font-weight: bold\"> threshold </span>┃\n",
       "┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.0833 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1       </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.0      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.01            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.9555    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.25   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 2       </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.5      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.02            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.9248    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4048 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 7       </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1429   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.07            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.8177    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4074 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 8       </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.25     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.08            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.8099    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4091 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 10      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.2      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1             </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.7784    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4119 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 11      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.2727   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.11            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.7746    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.3937 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 12      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.25     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.12            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.765     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.3997 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 13      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.3077   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.13            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.7606    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4279 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 29      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1379   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.29            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.6927    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4341 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 30      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1667   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.3             </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.6805    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4508 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 35      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1429   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.35            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.6213    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4537 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 36      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1667   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.36            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.6114    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4842 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 49      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1224   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.49            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.5561    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4932 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 50      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.14     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.5             </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.5553    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4629 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 61      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.1148   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.61            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.5011    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4639 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 62      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.129    </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.62            </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4988    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.4507 </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 100     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.08     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1.0             </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.0206    </span>│\n",
       "└────────┴─────────┴──────────┴─────────────────┴───────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mf1    \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mthreshold\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.0833\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1      \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.0     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.01           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.9555   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.25  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m2      \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.5     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.02           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.9248   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4048\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m7      \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1429  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.07           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.8177   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4074\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m8      \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.25    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.08           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.8099   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4091\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m10     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.2     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1            \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.7784   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4119\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m11     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.2727  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.11           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.7746   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.3937\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m12     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.25    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.12           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.765    \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.3997\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m13     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.3077  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.13           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.7606   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4279\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m29     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1379  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.29           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.6927   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4341\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m30     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1667  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.3            \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.6805   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4508\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m35     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1429  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.35           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.6213   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4537\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m36     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1667  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.36           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.6114   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4842\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m49     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1224  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.49           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.5561   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4932\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m50     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.14    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.5            \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.5553   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4629\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m61     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.1148  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.61           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.5011   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4639\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m62     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.129   \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.62           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.4988   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.4507\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m100    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.08    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1.0            \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.0206   \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└────────┴─────────┴──────────┴─────────────────┴───────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Total number of failures: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Total number of failures: \u001b[1;36m0\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds = agent.run(ds, max_items=100)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "refuel",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
